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This study examined relationships between the after-school literacy activities and test per- 
formance of students who passed and students who failed the 2003 administration of the 
Ontario Secondary School Literacy Test. It included both students who have been in the 
Canadian school system for most of their school lives and mostly use English as a first lan- 
guage (LI), and students who have recently entered the Canadian school system and use 
English as a second language (L2). This study demonstrates the relatively small yet com- 
plex relationship between the test performance of these different groups of students on this 
literacy test and their after-school reading and writing activities. 
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Cet article analyse les activites de litteratie parascolaires des eleves et les resultats qu'ils 
ont obtenus lors du Test provincial de competences linguistiques administre en Ontario en 
2003. L'etude porte a la fois sur des eleves qui ont ete dans le systeme scolaire canadien 
pour la majeure partie de leur vie et dont la langue premiere (LI) est l'anglais et sur des 
eleves qui sont entres recemment dans le systeme scolaire canadien et pour qui l'anglais est 
une langue seconde (L2). L' analyse devoile un lien relativement tenu mais neanmoins 
complexe entre le rendement de ces divers groupes d'eleves dans le test provincial de 
competences linguistiques et leurs activites de litteratie - lecture et production d'ecrits - 
parascolaires. 
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Large-scale testing has become increasingly common in public education 
as a means to ensure students are attaining fundamental educational 
outcomes. Tests are a differentiating ritual for students, and the conse- 
quences for students can vary dramatically depending on their perform- 
ance. As Stobart (2003) notes, "testing is never a neutral process and 
always has consequences" (p. 140). The Ontario Secondary School Liter- 
acy Test (OSSLT), a high school graduation requirement in Ontario, pro- 
vides a vivid example of a large-scale test that has an impact on students 
both significantly and differentially (Cheng, Klinger, & Zheng, 2007; 
Luce-Kapler & Klinger, 2005; Sider, 2003; Stroud & Cheng, 2002). Suc- 
cessful completion of the OSSLT, or the Ontario Secondary School Lit- 
eracy Course (OSSLC), a specially developed literacy course, is a re- 
quirement for graduation in Ontario. 

Large-scale high-stakes literacy testing like the OSSLT can be partic- 
ularly problematic for vulnerable groups of students who are second 
language learners or who have little formal education in English, the lan- 
guage in which literacy is being tested. These second language students 
are estimated to be from 20 to 50 per cent of the general student popula- 
tion in urban K-12 school systems across Canada (Roessingh, 1999). On- 
tario secondary schools in particular have seen the greatest increase of 
these students over the past 20 years. Simultaneously, large-scale 
achievement testing is increasingly used to measure and ensure student 
competency or provide system accountability (Firestone, Mayrowetz, & 
Fairman, 1998; Ryan, 2002). As evidenced by these second language stu- 
dents' low success and high deferral rates on the OSSLT (Education 
Quality and Accountability Office, 2002, 2004, 2007, 2008), many of these 
students are struggling as they attempt to quickly obtain and demon- 
strate fundamental English literacy skills. Given the relative importance 
of the OSSLT for graduation, it is important to identify and understand 
those factors related to their test performance. 

Although the goal of testing programs is largely to measure stu- 
dents' knowledge and skills of the domain being tested, previous re- 
search has demonstrated that students' test performance may not simply 
demonstrate their ability in the specified domain (Bachman, 1990; Kun- 
nan, 1995). The OSSLT is purported to measure students' essential read- 
ing and writing skills that they must attain by the end of grade 9 (EQAO, 
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2007). Variations in student performance on this test should not occur 
because of factors irrelevant to the construct the test intends to measure 
(e.g., Messick, 1989, 1996). Nonetheless, test performance will, in most 
cases, interact with students' personal attributes, personal experiences, 
and family background along with the factors derived from the test it- 
self, for example, test format. As an example, learning or knowledge is 
heavily dependent on social interaction (Bandura, 1986). Bandura's social 
cognitive theory postulates that "human functioning is explained in 
terms of a model of triadic reciprocality in which behaviour, cognitive 
and other personal factors, and environmental events all operate as inter- 
acting determinants of each other" (p. 18). Further, socio-cognitive fac- 
tors have been shown to influence test performance (Tarone & Bigelow, 
2005; Tarone, 2007). 

Although there is increased interest in the testing of second language 
learners (e.g., Abedi, 2004; Bishop & Mane, 2001; Cheng, 1998; Fox, 2003; 
Hakuta & Beatty, 2000; Horn, 2003), these studies have examined second 
language student populations alone, ignoring the broad context of the 
entire student population. Also researchers exploring the social and cog- 
nitive factors associated with test performance, including home lang- 
uage, computer ownership, and after-school activities have not consid- 
ered the potentially differential contributions of these factors on test per- 
formance of sub-populations of students (see Kunnan, 1998). Although 
one would expect to find relationships between students' performance 
on a literacy test such as the OSSLT and the types and frequency of liter- 
acy-related activities in which these students engage outside school, it is 
not clear if this relationship is consistent across first and second language 
learners. 

Given the potential importance of the interacting factors noted 
above, this study examined the after-school literacy activities, computer 
ownership and activities, and test performance of students who passed 
and failed the 2003 administration of the OSSLT. Further, the study in- 
cluded both students who were in the Canadian school system for most 
of their school life and used English as a first language (LI), and students 
who recently entered the Canadian school system and mostly used Eng- 
lish as a second language (L2). The purpose of the study was to compare 
not only the test response patterns of students having different personal 
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characteristics related to their language background and having passed 
or failed the test, but also the after-school reading, writing, and computer 
activities of these groups of students. The study also further examined 
the relationship between their test performance on the OSSLT and their 
after-school literacy activities. 

Why Study L2 Students in Comparison with LI Students? 

Large-scale, high-stakes literacy testing is particularly problematic for 
vulnerable groups of students who are second language learners, or who 
have had little formal education in the language being tested. Currently, 
an increasing numbers of such students in Ontario secondary schools 
bring an increasingly linguistic and cultural diversity to their learning. 
The Ontario curriculum (Ministry of Education and Training, 1999) and 
the Education Quality and Accountability Office 1 (EQAO) further distin- 
guish these students as English as a Second Language/English Language 
Development (ESL/ELD) students. ESL students are those who have 
recently arrived in Ontario schools "from another country with a first 
language other than English and with little or no previous experience 
with English." They are the students who may be proficient users of their 
own language(s). ELD students are "those who arrive in Canada from 
countries where English is the first language but other varieties of Eng- 
lish are in common use" (EQAO, 2002, p. 7) or those who have had little 
formal education in English. Two distinctions are acknowledged in this 
definition. The first is whether English is a student's first language. The 
second is whether a student has previously received formal schooling 
whether a student has previously received formal schooling before enter- 
ing the Ontario system. In our study, these two categories were treated 
and coded as one category termed as ESL/ELD students and based on 
the data we obtained from EQAO. 2 


1 The Education Quality and Accountability Office (EQAO) develops, distributes, collects, 
and scores the OSSLT. 

2 The newest policy and procedure document - English Language Learners: ESL and ELD 
programs and services (Ministry of Education, 2007) - redefines these students as English 
Language Learners (ELL). This category is further categorized into Canadian-born Eng- 
lish Language Learners and Newcomers from other countries (for details see 
http://www.edu.gov.on.ca/eng/document/ esleldprograms/ f. 
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To study these ESL/ELD students within the broad context of the 
entirety of the population of students taking the OSSLT, we included a 
sample from the population of students who wrote the OSSLT as a base- 
line for comparison. We categorized these students (test-takers) as non- 
ESL/ELD students in our study. These are students who have been in the 
Canadian school system for most of their school lives and mostly use 
English as a first language (LI). This category may contain students who 
have a first language other than English, but have been in Canada for a 
number of years and their teachers and schools do not regard them as 
ESL/ELD students. ESL/ELD or non-ESL/ELD status was determined by 
the EQAO's Student Information Form 3 that the students' schools com- 
pleted. 

In sum, this study, which involves a comparison of students with 
different characteristics (including language and schooling background), 
examines the relationship between test performance of these different 
groups of students on this literacy test and specific socio-cognitive fac- 
tors such as after-school reading and writing activities. 

LITERACY ACTIVITIES AND STUDENT LEARNING 

The study of literacy and its contribution to student learning have been 
well documented in education (e.g., Montigny, Kelly, & Jones, 1991; 
Willms, 1997). Not surprisingly, literacy development is linked to en- 
gagement in literacy-related activities (Early, 1992; Emmitt, Komesaroff, 
& Pollock, 2006; Montigny et al., 1991; Tarone & Bigelow, 2005; Tarone, 
2007). Hence, it is important to examine the contributions of extra- 
curricular and independent literacy activities and computer use to stu- 
dents' learning, especially for those students who have less formal expo- 
sure to the language being tested. 

Empirical studies suggest the extent of knowledge and exposure to 
English literacy materials are key factors in language development (Em- 
mitt, et al., 2006; Tarone & Bigelow, 2005; Tarone, 2007). Similarly, both 
the time devoted to and the types of after-school English reading and 
writing activities have been linked to literacy achievement in English 
(Early, 1992; Montigny et al., 1991). Krashen (1985, 2007) maintains that 


3 For details see http://www.ontla.on.ca/librarv/repository/mon/6000/ 10314940.pdf 
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extensive comprehensible input in terms of reading in English is crucial 
to English language acquisition. Caldwell and Game (2000) claim the 
best predictor of reading achievement is determined by the amount of 
time students spend on independent reading. More specifically, they 
found that at least one third of students' vocabulary growth could be 
accounted for by after-school independent reading because it helped to 
provide students with a wide range of background knowledge, and in 
turn, enhanced their reading achievement on tests. 

Similarly, extensive after-school reading helps improve the level of 
reading proficiency in a global sense, reinforcing the acquisition of 
grammar, vocabulary, and rhetorical structure, as well as increasing stu- 
dents' general knowledge base (Gradman & Hanania, 1991). Therefore, 
students who read extensively out of class attained higher levels of pro- 
ficiency than those with less after-school reading. Although the literature 
abounds with studies emphasizing the importance of the length and ex- 
posure to printed materials in English, we found no literature that ex- 
amined the relative contributions of different literacy materials to lit- 
eracy test performance. 

Computer Literacy 

Computer technology has ushered in a new era of mass media in stu- 
dents' learning (Wartella & Jennings, 2000). It is no longer reasonable to 
discuss students' academic development without mentioning the influ- 
ence of computer use. Computer technology, which has become more 
accessible to students, has become a leading form of communication for 
them (Liu, Moore, Graham, & Lee, 2002). The increasing amount of time 
students spend on home computers raises questions of how computer 
use might influence their academic literary development (Subrahman- 
yam. Kraut, Greenfield, & Gross, 2000). Lee (1997) points out that e-mails 
provide students with communicative opportunities for collaborative 
learning and create a relatively non-threatening atmosphere for students 
to express, negotiate, and interpret meaning within a meaningful con- 
text, which in turn contributes positively to their writing achievement on 
tests. Pennington (2003) claims that students' involvement in computer- 
related literacy activities positively promotes their attitudes. Both the 
quantity and quality of their literacy activities via Internet may positive- 
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ly influence, for example, writing or reading for extended periods of time 
and forming better revising behaviours. 

METHOD 

The Ontario Secondary School Literacy Test (OSSLT) was developed by 
the Education Quality and Accountability Office (EQAO) in Ontario, 
Canada. All students in public and private secondary schools who are 
working toward an Ontario Secondary School Diploma must complete 
the OSSLT, or the Ontario Secondary School Literacy Course (OSSLC) - 
for those who have been eligible to write the OSSLT at least twice and 
have been unsuccessful at least once. Because of the large geographic 
size of Ontario and variability in physical structures (urban and rural 
locations), education is expected to meet the needs of a very diverse stu- 
dent population. Since its trial run in 2000, seven administrations of the 
OSSLT have occurred by the writing of this article (February 2002, Octo- 
ber 2002, October 2003, October 2004, March 2006, March 2007, and 
March 2008). The data reported in this article are from the October 2003 
OSSLT administration. Compared with the overall population of stu- 
dents who write the OSSLT, ESL/ELD students show a much lower pass- 
ing rate. For example, in the 2003 test administration, the passing rates 
were 77 per cent for non-ESL/ELD students versus 42 per cent for 
ESL/ELD (EQAO, 2004) and the respective rates were 84 per cent versus 
52 per cent on the 2007 administration (EQAO, 2007). 

Since the 2004 administration, the format of the OSSLT has been 
changing, shortening the test and combining the reading and writing 
components. Hence, the 2003 test differs from the test currently in use, 
but the test constructs remain similar. The 2003 form of the OSSLT con- 
sisted of a reading and a writing component. The reading component 
had 12 short passages within three EQAO defined text types: information- 
al 4 (50%), graphic (25%), and narrative (25%). The passages included fic- 
tional, non-fictional, and graphical texts (e.g., graphs, schedules), al- 
though the actual passages differed across administrations. In response 
to the passages, the students demonstrated their understanding and 


4 Facets of the OSSLT item formats, text types, skills, and strategies of reading and the four 
writing tasks are italicized in this article as they are defined by the EQAO. 
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comprehension using test items with three different formats: multiple- 
choice (MC; 40%), constructed response (CR; 35%), and constructed response 
with explanations (CRE; 25%). The CR items required a short answer and 
the CRE items required a written explanation, that is, the students had to 
justify or explain the thinking behind their answers. The test items also 
measured three reading skills: (a) direct understanding, (b) indirect under- 
standing, and (c) connections, and four reading strategies: (a) vocabulary, 
(b) syntax, (c) organization, and (d) graphical features. The EQAO defined 
these skills and strategies, based on the learning expectations of the On- 
tario curriculum. The writing component consisted of four writing tasks: 
(a) a summary, (b) a series of paragraphs expressing an opinion, (c) a news 
report, and (d) an information paragraph. The purpose and audience for 
each task was provided (e.g., to report on an event for the readers of a 
newspaper). Students were also provided with guidelines regarding the 
length and methods to structure each writing sample. 

The OSSLT is administered by each Ontario secondary school. The 
completed tests are sent to the EQAO where they are marked centrally 
by teachers who were trained as markers. The MC and CR items on the 
reading component are scored on a 2-point (0, 2) scale and the CRE items 
are scored using item-specific scoring rubrics on a 3-point scale (2 points 
for correct, 1 point for partly correct, or 0 for incorrect). Each of the four 
writing tasks is scored using a 4-point (1-4, but allocated 10, 20, 35 and 45 
marks) 5 scoring scale having specific performance descriptors. A score of 
0 is given for responses classified as "blank/illegible and irrelevant con- 
tent/off-task." A score of 4 is given to the highest quality written res- 
ponses. 

At the same time as the OSSLT administration, students complete a 
questionnaire, designed by the EQAO that is composed of four major 
parts: after-school reading activities, after-school writing activities, home 
computer use, and home language background. The data are measured 
on either a dichotomous or an ordinal scale (see Appendix A). 


5 A Level 1-4 system is used in all Ontario student report cards for school academic 
achievement. 
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Research Questions 

This study examined the relationship between students' after-school 
reading, writing, and computer activities and their OSSLT performance. 
In particular, we were interested in potential differences that could be 
attributed to students' characteristics, for example, ESL/ELD status and 
OSSLT performance. Consequently, we studied four groups of students, 
LI students (non-ESL/ELD) who passed and failed the 2003 OSSLT and 
L2 (ESL/ELD) students who passed and failed the 2003 OSSLT. We there- 
fore termed the four groups as non-ESL/ELD pass, non-ESL/ELD fail, 
ESL/ELD pass, and ESL/ELD fail students. The study addressed three 
research questions: 

1. How did the four groups of students perform on the different fac- 
ets/formats of reading and the different writing tasks of the OSSLT? 

2. What were the responses from the four groups of students to the 
EQAO questionnaire regarding after-school reading and writing ac- 
tivities? 

3. What were the relationships between the four groups of students' 
after-school reading and writing activities and their test performance 
on the OSSLT? 

Datasets 

Three datasets containing student performance results on the October 
2003 OSSLT administration and their responses to the EQAO question- 
naire were obtained from the EQAO. The first contained a sample of 
non-ESL/ELD students who passed both components of the OSSLT (n = 
5,000), the second had a sample of non-ESL/ELD students who failed 
both components of the OSSLT (n = 5,000), and the third dataset was the 
population of ESL/ELD students who completed the OSSLT (n = 4,311). 
We subsequently formed four groups of students from these three data- 
sets. In separating the ESL/ELD students, the ESL/ELD pass group con- 
tained those students who successfully completed both the reading and 
writing components; the ESL/ELD fail group included those who failed 
both components. The cut score was 125 out of 200 marks for reading, 
and 100 out of 180 marks for writing. This procedure resulted in a sam- 
ple of 1,720 students in the ESL/ELD pass group, and 1,261 in the 
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ESL/ELD fail group. We then combined the data from the four groups of 
students to form a new overall dataset distinguishing the four groups. In 
addition, students' responses to the EQAO questionnaire were linked to 
their test performance. 

Data Analyses 

Data were analyzed in three stages to answer the three research ques- 
tions. First, we compared the four groups on total reading test scores and 
on reading sub-scores based on item formats, text types, and skills and 
strategies. We examined group comparisons on the writing tasks both at 
the task and total score level. Descriptive statistics included means and 
standard deviations. We conducted analysis of variance (ANOVA) pro- 
cedures of the reading sub-scores to determine significant performance 
differences among the four groups. Where we found significant differ- 
ences, we conducted post hoc procedures using Tamhane's method. 
Because of the number of similar ANOVA analyses, we set a to .01 to 
reduce the potential for Type I error. Chi-square tests were performed to 
compare the writing task performance among the four groups. 

Second, we calculated descriptive statistics for the EQAO question- 
naire items for the four groups. Specifically, these items examined after- 
school reading and writing activities, reading hours per week, writing 
hours per week, and weekly home computer use. In the case of computer 
use, we conducted analyses to determine group differences in the own- 
ing of a home computer and also to determine group differences in com- 
puter use among students who did own a computer. 

Third, we explored relationships between after-school literacy activ- 
ities and reading and writing performance using multiple regression. We 
considered the non-ESL/ELD pass group as the reference group, and 
employed a stepwise method to create a predictive model for this partic- 
ular group. Two steps were involved in conducting the multiple regres- 
sions for the other three groups. First, based on the significant variables 
obtained from the reference group, we entered the same set of variables 
in the same order of importance for each of the other groups. Next, using 
the stepwise method, we added all non-significant variables from the 
reference model to the new models. The purpose of this design was to 
compare the predictive model among the four groups. This procedure 
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also enabled the identification of any unique variable(s) that were not 
significant in the reference group but were significant in other groups. 

RESULTS 

We present the results of this study in three sections, corresponding to 
the three research questions. First, the average students' test scores for 
each group are compared. Second, the students' questionnaire responses 
are presented. Third, the predictive variables from the after-school liter- 
acy activities on test performance are identified and compared across 
groups. 

Test Performance 

Table 1 provides the comparisons of the total reading scores and the 
reading sub-scores based on item formats, text types, skills, and strate- 
gies. The first vertical panel contains the results for the non-ESL/ELD 
groups, and the second vertical panel contains the results for the 
ESL/ELD groups. Descriptive statistics provide the overall distributions 
of total reading scores and reading sub-scores and illustrate the relative 
performance differences among the four groups. 

In terms of item format, the analyses demonstrate that both ESL/ 
ELD and non-ESL/ELD students perform relatively better on the mul- 
tiple-choice (MC) test items; while the constructed response with explan- 
ations (CRE) was the most difficult item format for all four groups. For 
example, the non-ESL/ELD pass group achieved an average score of 80.8 
per cent on the MC items, 80.7 per cent on the CR items, and 73.1 per 
cent on the CRE items. The ESL/ELD pass group averaged 73.4 per cent, 
75.7 per cent, and 68.4 per cent on each set of sub-scores. In contrast, the 
non-ESL/ELD fail group obtained an average of 52 per cent, 46.8 per 
cent, 39.9 per cent, and the ESL/ELD fail group obtained an average of 
46.4 per cent, 38.3 per cent, and 32.5 per cent on these sub-scores. The 
standard deviations in the scores demonstrate that the ESL/ELD fail 
group had the most varied scores, followed by the non-ESL/ELD fail 
group. The two pass groups had relatively smaller variations in group 


score. 
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Table 1 

Percentage Scores for Reading Item Formats, Text Types, Skills and Strategies 


Non-ESL/ELD ESL/ELD 

Pass Fail Pass Fail 



O/ 

/o 

SD 

O/ 

/o 

SD 

O/ 

/o 

SD 

O/ 

/o 

SD 

Reading Item Format: 

Multiple choice (MC) 
Constructed 

80.8 

9.3 

52.0 

11.5 

73.4 

9.0 

46.4 

11.5 

Responses (CR) 
CR with 

80.7 

9.3 

46.8 

13.8 

75.7 

8.6 

38.3 

15.6 

Explanations (CRE) 

73.1 

10.9 

39.9 

14.0 

68.4 

10.4 

32.5 

15.3 

Reading Skills: 

Understanding 
directly stated ideas 

81.9 

9.7 

51.2 

12.0 

76.7 

8.3 

48.2 

13.0 

Understanding indi- 
rectly stated ideas 

79.7 

8.7 

48.3 

12.3 

73.5 

8.8 

39.6 

13.0 

Making connections 

73.6 

10.7 

40.2 

13.5 

67.5 

10.5 

31.2 

14.5 

Reading Text Types: 

Informational 

77.2 

10.0 

42.2 

11.9 

70.4 

9.3 

36.1 

12.0 

Graphic 

82.7 

9.4 

52.5 

14.2 

80.2 

8.9 

47.6 

15.9 

Narrative 

77.9 

9.4 

50.0 

13.5 

70.7 

9.4 

39.8 

14.2 

Reading Strategies: 

Vocabulary 

76.2 

14.9 

37.3 

15.0 

59.4 

15.5 

27.7 

13.2 

Syntax 

72.5 

13.0 

37.6 

14.7 

65.2 

13.0 

31.2 

14.2 

Organization 

75.9 

12.2 

40.7 

13.6 

71.0 

12.2 

36.1 

13.8 

Graphic features 

82.8 

10.8 

54.1 

17.6 

82.6 

10.6 

49.7 

19.3 

Total scores 

157.6 

16.1 

94.3 

21.8 

145.9 

14.5 

80.2 

24.0 


Note: Percentage score= mean /subtotal score*100 

Percentage SD= SD of the mean/subtotal score*100 
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Similar performance differences were found among the four groups 
on the reading constructs of text types, skills, and strategies. The informa- 
tion text type, the making connections skill, and the syntax strategy were 
the most difficult for the non-ESL/ELD pass group. These patterns for 
text type and reading skills were consistent across groups. In contrast, 
vocabulary appeared to be the most difficult strategy for the other three 
groups. Overall, the results highlight notable similarities and differences 
between ESL/ELD and non-ESL/ELD groups. 

We performed ANOVA analyses of the reading sub-scores on the 
reading constructs. Homogeneity of variances was first tested using the 
Levene's statistics. The significant Levene's statistics for all the reading 
sub-scores indicated unequal variances across groups. Hence, multiple 
comparisons were completed using Tamhane's procedure. The non- 
ESL/ELD and the ESL/ELD pass groups were not statistically different 
only in graphic features (p >.01). All other sub-scores comparisons among 
the four-groups were statistically different. 

We also compared writing task performance among the four groups 
of students. Table 2 contains the proportion of students who obtained 
each score point and mean scores on the four writing tasks. Not surpris- 
ingly, larger proportions of students in the two fail groups obtained low- 
er scores (0, 1, or 2) than those in the pass groups, especially scores of 
zero or two. The ESL/ELD pass group had the highest proportion of stu- 
dents obtaining a score of three, but had proportionately fewer scores of 
four than the non-ESL/ELD pass group. With the exception of the sum- 
mary writing task, the rank order of the mean writing scores were non- 
ESL/ELD pass, ESL/ELD pass, non-ESL/ELD fail, and ESL/ELD fail. 
Unexpectedly, the ESL/ELD pass group outperformed the non-ESL/ELD 
pass group and the ESL/ELD fail group outperformed the non-ESL/ELD 
fail group on the summary writing task. Chi-Square tests identified signif- 
icant differences across the four groups on all the writing tasks 
(P <-01). 
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Table 2 

Distribution of Students' Performance on the Writing Tasks 


Writing 

Non-ESL/ELD 

ESL/ELD 

Writing 

Non-ESL/ELD 

ESL/ELD 

task 

(%) 


(%) 


task 

(%) 


(%) 



Pass 

Fail 

Pass 

Fail 


Pass 

Fail 

Pass 

Fail 

Summary 





News report 





0 points * 

2.1 

44.0 

.6 

29.1 

0 point 

2.6 

43.1 

2.8 

57.4 

1 point 

5.1 

21.7 

3.6 

17.6 

1 point 

0 

.2 

.1 

.4 

2 points 

22.1 

23.2 

22.3 

30.2 

2 points 

6.6 

34.3 

14.7 

34.5 

3 points 

55.3 

10.8 

58.5 

21.6 

3 points 

64.7 

21.8 

65.8 

7.4 

4 points 

15.5 

.3 

15.0 

1.4 

4 points 

26.1 

.6 

16.7 

.3 

Mean 

31.2 

10.7 

32.0 

16.0 

Mean 

35.7 

14.8 

33.5 

9.7 

Paragraphs expressing an opinion 



Information paragraph 




0 point 

.1 

11.2 

.1 

17.0 

0 point 

.2 

21.3 

.3 

25.5 

1 point 

.4 

22.2 

.5 

21.6 

1 point 

3.3 

18.4 

4.0 

20.6 

2 points 

4.6 

36.9 

5.7 

35.1 

2 points 

27.6 

47.7 

29.6 

44.8 

3 points 

58.6 

28.5 

65.7 

25.8 

3 points 

52.0 

12.2 

54.2 

9.0 

4 points 

36.3 

1.2 

28.0 

.5 

4 points 

16.9 

.5 

11.9 

.2 

Mean 

37.8 

20.1 

36.8 

18.4 

Mean 

31.7 

15.9 

30.6 

14.2 


Note 1: 0 points are awarded for Blank/ Illegible or Irrelevant Content or if the response is 'Off- task'. 

Note 2: scores for each task is weighted according to the following weighting scheme: 0 is converted to 0, 1 point is converted to 10 
points, 2 points is converted to 20 points, 3 points is converted to 35 points, and 4 points is converted to 45 points. 
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Questionnaire Responses 

The students' responses to the EQAO questionnaire items are summar- 
ized in Table 3. The first horizontal panel of the table contains the results 
of the dichotomous after-school literacy items related to types of reading 
in English. Not surprisingly, the two pass groups reported a greater like- 
lihood to engage in reading activities than the two fail groups. The most 
common reading type was reading on the Internet for three of the groups 
(82.3% for the non-ESL/ELD pass group, 76.3% for the ESL/ELD pass 
group, 58.8% for the ESL/ELD fail group). For the non-ESL/ELD fail 
group, magazine reading was most commonly reported (63.7%) with read- 
ing on the Internet being second (57.6%). The next two frequently reported 
reading types for the two pass groups were reading magazines and novels 
(76.2% and 61.5% for the non-ESL/ELD pass group, 51.8% and 59.8% for 
the ESL/ELD pass group). Reading on the Internet was followed by poetry 
for the non-ESL/ELD fail group, while reading magazines and newspapers 
were the next most common for the ESL/ELD fail group. Manuals and 
religious reading were the least reported types for all four groups. 

Apart from the most reported and least reported reading types, the 
non-ESL/ELD fail group reported proportionately less reading of news- 
papers (35.4%) than the ESL/ELD fail group (41.4%). Both the ESL/ELD 
pass and ESL/ELD fail groups had higher proportions of reading religious 
materials (14.8%, 14.3%) than both non-ESL/ELD groups (13.4%, 10.1%), 
an intriguing result worth further exploration. 

Students were also asked by the questionnaire to identify the English 
language reading materials they possessed at home. The majority of the 
students in the two pass groups reported that they owned all four types 
listed in the questionnaire. The possession rates for these materials were 
much lower in the two fail groups. 

The third horizontal panel of Table 3 contains the items related to 
writing in English. Other than writing e-mails, which the majority of stu- 
dents from all four groups reported that they did the most, the other 
forms of after-school writing, for example, writing notes, songs, or stories, 
were less commonly conducted across groups. These frequencies indi- 
cate that students were more likely to be involved in reading activities as 
opposed to writing activities after school, regardless of their group. 
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Table 3 

Descriptive Statistics of After-school Literacy Activities 


Questionnaire items Non-ESL/ELD (%) ESL/ELD (%) 

Pass Fail Pass Fail 


Types of reading in English after school most weeks 


Non-fiction 

24.8 

20.4 

20.9 

16.3 

Comics 

34.0 

30.9 

28.8 

24.0 

Internet 

82.3 

57.6 

76.3 

58.8 

Letters 

30.3 

29.9 

25.1 

28.0 

Magazines 

76.2 

63.7 

51.8 

48.9 

Manuals 

22.8 

14.5 

18.8 

8.6 

Newspapers 

46.8 

35.4 

43.2 

41.4 

Novels 

61.5 

33.6 

59.8 

40.0 

Poetry 

42.6 

35.9 

31.8 

28.5 

Religious 

13.4 

10.1 

14.8 

14.3 

English-language reading materials at home 



Dictionaries 

92.4 

64.5 

85.8 

61.1 

Books 

93.8 

71.5 

80.0 

65.0 

Newspapers 

87.3 

61.1 

66.3 

45.3 

Magazines 

88.7 

70.9 

55.8 

47.4 

Types of writing in 

English after school most weeks 


E-mail 

91.0 

75.1 

85.9 

71.2 

Letters 

36.2 

31.5 

31.0 

32.5 

Notes 

35.0 

24.8 

30.9 

24.7 

Songs 

31.3 

34.3 

20.6 

25.1 

Stories 

18.5 

15.4 

14.7 

19.7 

Work-related 

35.9 

27.1 

38.5 

28.4 
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We calculated frequencies of reading and writing hours per week in 
English after school and identified group differences. These hours ex- 
cluded homework time. Among the two pass groups, a large proportion 
of students reported spending more than three hours reading (40% for 
the non-ESL/ELD pass group, 34.4% for the ESL/ELD pass group). In 
comparison, fewer students in the fail groups reported this level of read- 
ing: 15.1 per cent for the non-ESL/ELD fail group, and 20 per cent for the 
ESL/ELD fail group. In terms of writing, 29.2 per cent of the non- 
ESL/ELD pass group and 23.3 per cent of the ESL/ELD pass group re- 
ported spending more than three hours of weekly writing. A smaller 
percentage of the fail groups reported this amount of time writing (18.6% 
for non-ESL/ELD fail group, 18.5% for ESL/ELD fail group). 

The examination of computer use highlights that the two pass 
groups also reported having higher proportions of computer possession 
at home than the two fail groups (98% non-ESL/ELD pass, 89.8% non- 
ESL/ELD fail, 97.3% ESL/ELD pass, and 90.2% ESL/ELD fail). Students in 
all four groups who had computers at home reported frequent use for 
school work. Both the ESL/ELD pass and the ESL/ELD fail groups had 
higher percentages than the non-ESL/ELD groups in the two categories 
of frequent computer use (once or twice a week, or almost every day). 
The pass groups reported that they read on the Internet and wrote e- 
mails frequently: 83.2 per cent and 92 per cent for the non-ESL/ELD pass 
group; 78.6 per cent and 88.1 per cent for the ESL/ELD pass group. The 
fail groups made less use of the computer for these purposes, although 
more than half of these students engaged in these computer activities 
(ranging from 61.6% to 80.9%). In addition, the two fail groups were less 
likely to use computers even if they had computers at home. 

After-School Literacy Activities and Test Performance 

Table 4 contains those after-school literacy activities that predicted stu- 
dents' reading and writing scores on the OSSLT. The non-ESL/ELD pass 
group was the reference group, and eight variables were significant pre- 
dictors of reading scores for this group: (a) reading hours per week, (b) read- 
ing novels, (c) having a dictionary, (d) writing songs, (e) reading on the Inter- 
net, (f) reading letters, (g) home computer use, and (h) having books. Writing 
songs and reading letters were negative predictors, suggesting that after 
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accounting for the other variables, increasing amounts of these two activ- 
ities were associated with lower reading scores. Seven variables were 
significant predictors of the writing scores: (a) reading novels, (b) home 
computer use, (c) reading hours per week, (d) reading on the Internet, (e) writ- 
ing songs, (f) reading the newspaper, and (g) having dictionaries, with writing 
songs being a negative predictor. The variances explained by these var- 
iables were small: 14 per cent for reading, and 6 per cent for writing. 
Although small, these values were larger than those found for the other 
three groups in which the accounted variance was 10 per cent or less for 
reading and less than 5 per cent for writing. Any unique variable(s) that 
were not significant in the non-ESL/ELD pass group, but were significant 
in other groups, are indicated in bold in Table 4. For the non-ESL/ELD 
fail group, writing songs and home computer use were not significant 
(p>.01). The other six variables were significant, along with writing stories, 
reading religious materials, and having magazines. For the non-ESL/ELD fail 
group, writing stories and reading religious materials were negative predic- 
tors in the presence of the other variables. In the writing model for the 
non-ESL/ELD fail group, the only significant predictors were home com- 
puter use, reading on the Internet, and having a dictionary along with the 
additional variable, having English books at home. Reading letters, home 
computer use, and having English books were not significant predictors of 
reading performance for the ESL/ELD pass group. In addition to the five 
remaining significant predictors, reading manuals and writing letters were 
also significant predictors of their reading scores. The only significant 
predictor of writing scores was the number of reading hours/week. Few 
variables were significant predictors of reading and writing for the 
ESL/ELD fail group. Both reading novels and having a dictionary positively 
predicted reading and writing. No other significant predictors of reading 
or writing were found after accounting for these two variables. 

Based on the results, the predictors of reading and writing scores 
appear to vary for these different groups of students. Significant predic- 
tors for the non-ESL/ELD pass group were not necessarily predictive of 
performance for the other groups. Further, the variances explained by 
the significant variables in both reading and writing were small, decreas- 
ing from the non-ESL/ELD pass group to the non-ESL/ELD fail group, 
the ESL/ELD pass group and the ESL/ELD fail group. 
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Table 4 

Multiple Regression Results 


Reading 

P 

t 

Sig. 

R 2 

Writing 

P 

t 

Sig. 

R 2 

Non-ESL/ELD Pass 




.14 





.06 

Hrs Reading/week 

.22 

15.71 

.00 


Reading Novels 

.11 

7.42 

.00 


Reading Novels 

.15 

10.33 

.00 


Home Computer Use 

.11 

7.71 

.00 


Having a dictionary 

.09 

5.98 

.00 


Hrs Reading/week 

.08 

5.26 

.00 


Writing Songs 

-.08 

-5.78 

.00 


Reading Internet 

.06 

4.16 

.00 


Reading Internet 

.08 

5.91 

.00 


Writing Songs 

-.06 

-4.30 

.00 


Reading Letters 

-.08 

-6.12 

.00 


Reading Newspapers 

.05 

3.57 

.00 


Home Computer Use 

.06 

4.46 

.00 


Having a dictionary 

.050 

3.48 

.00 


Having books 

.06 

3.83 

.00 







Non-ESL/ELD Fail 




.10 





.03 

Hrs Reading/week 

.05 

3.63 

.00 


Reading Novels 

.01 

.32 

.75 


Reading Novels 

.05 

3.09 

.00 


Home Computer Use 

.10 

6.80 

.00 


Having a Dictionary 

.19 

12.53 

.00 


Hrs Reading/week 

-.02 

-1.06 

.29 


Writing songs 

.01 

.76 

.45 


Reading Internet 

.05 

3.58 

.00 


Reading Internet 

.07 

4.78 

.00 


Writing Songs 

-.00 

-.09 

.93 


Reading Letters 

-.07 

-4.92 

.00 


Reading Newspapers 

.01 

.73 

.47 


Home Computer Use 

.04 

2.42 

.02 


Having a Dictionary 

.07 

4.28 

.00 


Having books 

.06 

3.98 

.00 


Having Books 

.06 

3.66 

.00 


Writing Stories 

-.10 

-6.71 

.00 







Having Magazines 

.10 

6.58 

.00 







Reading Religious 

-.06 

-3.84 

.00 
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ESL/ELD Pass .08 .03 


Hrs Reading/week 

.13 

5.13 

.00 

Reading Novels 

.066 

2.63 

.01 

Reading Novels 

.11 

4.27 

.00 

Home Computer Use 

.062 

2.53 

.01 

Having a Dictionary 

.07 

2.99 

.00 

Hrs Reading/week 

.097 

3.87 

.00 

Writing songs 

-.07 

-2.97 

.00 

Reading Internet 

.043 

1.75 

.08 

Reading Internet 

.09 

3.62 

.00 

Writing Songs 

- 

-1.91 

.06 

Reading Letters 

-.03 

-.98 

.33 

Reading Newspapers 

- 

-.45 

.66 

Home Computer Use 

-.01 

-.27 

.79 

Having a Dictionary 

.040 

1.62 

.11 

Having books 

.05 

1.96 

.05 





Reading manuals 

.09 

3.83 

.00 





Writing letters 

-.10 

-3.67 

.00 





ESL/ELD Fail 



.07 





Hrs Reading/week 

.02 

.77 

.44 

Reading Novels 

.087 

2.902 

.00 

Reading Novels 

.11 

3.50 

.00 

Home Computer Use 

.042 

1.417 

.16 

Having a Dictionary 

.11 

3.79 

.00 

Hours Reading Per 

- 

-1.168 

.24 

Writing songs 

-.02 

-.58 

.56 

Reading Internet 

.023 

.752 

.45 

Reading Internet 

.09 

2.86 

.00 

Writing Songs 

.013 

.437 

.66 

Reading Letters 

-.06 

-1.99 

.05 

Reading Newspapers 

- 

-.336 

.74 

Home Computer Use 

.07 

2.43 

.02 

Having a Dictionary 

.092 

2.985 

.00 

Having books 

.03 

1.09 

.28 
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DISCUSSION 

Test Performance 

Overall, the reading performance of these groups of students exhibited 
the following patterns. Not surprisingly, the performance of the 
ESL/ELD pass group was significantly lower than the non-ESL/ELD pass 
group and the ESL/ELD fail group had significantly lower performance 
than the non-ESL/ELD fail group in the reading constructs (text types, 
reading skills, and strategies) and across test formats (see Table 1 and 
ANOVA results). This result suggests the OSSLT is more difficult for 
second language students compared with their first language counter- 
parts. However, the relative performance pattern was similar for all four 
groups: if one group achieved a relatively higher score on one facet of 
the test, the other three groups were also likely to have relatively higher 
scores on this facet. In this sense, there was consistency in test perform- 
ance across the four groups of students. 

Looking at the differences more closely, the largest difference be- 
tween the two pass groups was the reading strategy associated with 
vocabulary. With respect to the two fail groups, the narrative type of read- 
ing distinguished the groups the most, followed by the strategy asso- 
ciated with vocabulary. Hence ESL/ELD students did relatively poorer 
using vocabulary as compared to their non-ESL/ELD counterparts, sug- 
gesting the OSSLT is largely an English language proficiency test, i.e. a 
test primarily determines the level of English language, for ESL/ELD 
students. ESL/ ELD students need to develop more in-depth vocabulary 
to be able to perform better on the OSSLT. In addition, the relatively 
large difference in successfully comprehending narrative text type be- 
tween the two fail groups suggests that ESL/ELD students also need to 
read more nar-rative types of materials, in a sense, more word-intensive 
texts. 

Examining the writing performance patterns, the majority of the stu- 
dents' writing had obtained a score of three for both pass groups. In fact, 
more ESL/ELD pass students fell into this category than students in the 
non-ESL/ELD group, albeit this was largely due to the larger proportion 
of non-ESL/ELD students who obtained a score of four. Students in the 
two fail groups were much more likely to obtain a score of zero on their 
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writing tasks. The largest relative performance difference between the 
two pass groups was in the news report task. In addition, the summary 
writing scores had different rank orders across the four groups as com- 
pared with the other three writing tasks. For the summary writing task, 
the ESL/ELD pass group outperformed the non-ESL/ELD pass group, 
and the ESL/ELD fail group outperformed the non-ESL/ELD fail group. 
This result is unexpected but suggests ESL/ELD students have had com- 
parably more success developing their summary writing skills. However, 
it is hard to interpret the reasons behind this result within the scope of 
this study. 

After-School Literacy Activities 

Inconsistent patterns exist in the after-school reading activities of these 
groups of students. Students in the two fail groups generally reported 
much less engagement in after-school reading. The ESL/ELD students 
who passed the OSSLT reported higher engagement in reading than the 
two groups of students who failed the OSSLT but less reading than the 
non-ESL/ELD students who passed the OSSLT. Students in the two 
non-ESL/ELD groups were most likely to read on the Internet, magazines, 
and novels, although the relative order differed between the two groups. 
Students in the ESL/ELD groups were most likely to read on the Internet, 
magazines, and newspapers, in the same order of preference. Thus novels, 
which consist of mostly narrative text types and with more word- 
intensive vocabulary, were a relatively less common form of English lan- 
guage reading for ESL/ELD students. And this difference appears to be 
reflected in their OSSLT vocabulary related performance. Dictionaries 
and books were the most commonly found reading materials in the 
homes of students. The two groups who passed the OSSLT were more 
likely to have reading materials at home with the non-ESL/ELD pass 
group being the most likely group to have these materials at home - in- 
dicating a potential relationship between possessing reading materials 
and higher OSSLT reading performance. 

In terms of writing activities, e-mail was consistently the most fre- 
quently reported activity that all groups of students did in English after 
school. This finding supports the idea that computer technology has be- 
come increasingly more accessible to students (Liu et al., 2002), and stu- 
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dents are spending an increasing amount of time on computers during 
their leisure time (Subrahmanyam et al., 2000). 

Students who successfully completed the OSSLT spent a greater 
amount of their after-school time on reading and writing. Again the suc- 
cessful non-ESL/ELD students reported the greatest amount of after- 
school reading and writing. Although ESL/ELD students were less en- 
gaged in reading and writing in English, it is not clear whether and how 
much after-school literacy activities these students completed in their 
first language. 

Relationship Between Test Performance and After-School Literacy Activities 

The regression results indicate the set of after-school literacy activities 
and materials account for a relatively small proportion of the variance in 
students' reading and writing scores on the OSSLT. Further, the ac- 
counted variance is even smaller for second language and failing sub- 
groups as compared with the main population of non-ESL/ELD students 
who passed the OSSLT. Different predictors of after-school activities 
were associated with reading and writing OSSLT performance among 
the four groups. Eight literacy variables significantly predicted OSSLT 
reading performance and seven literacy variables predicted writing per- 
formance in the non-ESL/ELD pass group, albeit to a lesser extent. The 
results for the ESL/ELD pass group were slightly different for reading, 
with only five of the same eight variables predicting reading along with 
two others, reading manuals and writing letters. There were even fewer 
shared predictor variables for the two fail groups relative to the non- 
ESL/ELD pass group. These results highlight the relatively small albeit 
complex relationships among after-school literacy activities and stu- 
dents' test performance on the OSSLT. Further, this result indicates that 
other intervening factors likely influence students' test performance, in- 
cluding students' home language patterns and parents' income and their 
educational levels. Although data limitations prevented the inclusion of 
such factors in the current study, future research will endeavour to in- 
clude these and other factors together to further our examination of the 
complex relationships and contributions to students' test performance. 
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CONCLUSION 

Considering language acquisition and development is a social cognitive 
process, it seems crucial to integrate the social and cognitive dimensions 
to enhance researchers' understandings of language development (Lewis 
& Carpendale, 2004). Dewey (1938) argued that people learn from social 
experiences, whether positive or negative, and the accumulated exper- 
iences, in one way or another, influence or interact with potential future 
experiences. Our study of various factors and their relationship with 
students' test performance on the OSSLT has illustrated the complex pic- 
ture of such connections among different groups of students. 

Overall, ESL/ELD students who pass the OSSLT received lower 
scores than non-ESL/ELD students who passed the test and ESL/ELD 
students who failed the test obtained lower scores than non-ESL/ELD 
students who also failed the test. Nevertheless, the relatively consistent 
score patterns across the groups of students suggest that this particular 
literacy test did not have inherent structural imbalances while assessing 
the literacy skills of a diverse population. In contrast, the less consistent 
findings in the contribution of after-school reading and writing activities 
to students' test performance illustrate the differences in literacy and 
language development for these four different groups. Perhaps, targeted 
and differentiated literacy instruction with varying focuses is required in 
high schools for LI and L2 students. As an example, focused reading 
instruction for second language students could use more word-intensive 
narrative texts along with in-depth vocabulary development. This obser- 
vation is supported by this study but also by other studies (Cheng, et al., 
2007, Qian & Schedl, 2004, Tarone & Bigelow, 2005). It should be noted 
that categories of participants are limited by the data we obtained from 
EQAO. In particular, the ESL/ELD group has highly heterogeneous char- 
acteristics (Cumming, Hart, Corson, Labrie, & Cummins, 1993) and 
worthwhile to study further, and using other more naturalistic methods. 

Despite the complexity of the relationships, there is an overall pat- 
tern on the OSSLT between after-school reading and writing activities 
and students' test performance. The more reading and writing activities 
these students did and the more reading materials these students owned, 
the higher their literacy test performance on the test. And the more fre- 
quently the students used computers at home for school work, the high- 
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er their test performance. This conclusion partially addresses the issue 
posed by Subrahmanyam et al. (2000) regarding the increasing amount 
of time students are spending on computers at home and how such tech- 
nology makes a difference in literary learning. 
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APPENDIX A 

EQAO Student Questionnaire 


MAY-19-2005 16:07 EQAO 41632S6622 

■ Ontario Secondary School Literacy Test 

October 2003 

Student 
Questionnaire 

This questionnaire should take less than five minutes. We need to know about your background, so we can see how 
students with different experiences do on the test. Your responses will remain completely anonymous. All the 
questionnaire responses will be summarized and reported at the school, board and provincial levels. 

Read each question carefully. Fill In the circle that best describes your response. Make sure the circle ia entirely filled] 

If no answer seems exactly right, fill in the circle with the response closest to what you want to say. If you wish to change 
one of your answers, put an X through the incorrect answer and fill in the appropriate circle as shown below. 

• Final answer Incorrect answer 


Reading 

1 . Indicate the types of materials you read in English 
outside school most weeks (fill in all circles that apply)- 
O Non-fiction books, e.g.. biographies 
O Comics 

C Web sites, e-mail, chat messages 
C Letters 
O Magazines 
O Manuals, instructions 
O Newspapers 

O Novels, fiction, short stories 
O Poetry, song lyrics 
O Religious or spiritual writings 
Indicate the number of hours a week you read 
materials written in English, not including your 
homework, outside school (fill In one circle only). 

O one hour or less 

O more than one hour and less than three hours 
O more than three hours and less than five hours 
O five hours or more 

Indicate what English-language materials you have at 
home (fill in all circles that apply). 

O Dictionaries, encyclopedias (print or electronic) 

O Books 
O Newspapers 
O Magazines 
Writing 

4. Indicate the types of writing you do in English outside 
school most weeks (fill in all circles that apply). 

O E-mail messages, chat-room conversations 
O Letters, journals, diaries 
O Notes, directions, instructions 
O Song lyrics, poems 
O Stories, fiction 
O Work related writing 


2 . 


3. 


5. Indicate the number of hours a week you write 4 
English, not including your homework, outside 
(fill in one circle only). 

0 one hour or less 

O more than one hour and less than three hours] 

O more than three hours and less than five hour| 

0 five hours or more 

Home Computer Use 

6. Indicate how often you use a computer at home 
school work (fill In one circle only). 

G 1 don't have a computer at home. 

G 1 never or hardly ever use the computer for school 

work. 

0 1 use the computer once or twice a month for School 
work. 

O I use the computer once or twice a week for s4hool 
work. 

O I use the computer almost every day for schoijl 
work. 

Language 

7. a) Is English the first language you learned at h^me? 

OYes 
ONo 

b) What languages do you speak at home (fill irj 
circle only)? 

O Only or mostly English 
G Another language (or languages) as often 
English 

O Only or mostly another language (or other] 
languages) 

Thank you for completing this questionnaire) 
Please return it to the teacher administering the 


KStudsnt Questionnaire 




